Vision-Language Integration in Multimodal Video Transformers (Partially) Aligns with the Brain

Integrating information from multiple modalities is arguably one of the essential prerequisites for grounding artificial intelligence systems with an understanding of the real world. Recent advances in video transformers that jointly learn from vision, text, and sound over time have made some progress toward this goal, but the degree to which these models integrate information from modalities still remains unclear. In this work, we present a promising approach for probing a pre-trained multimodal video transformer model by leveraging neuroscientific evidence of multimodal information processing in the brain. Using brain recordings of participants watching a popular TV show, we analyze the effects of multi-modal connections and interactions in a pre-trained multi-modal video transformer on the alignment with uni- and multi-modal brain regions. We find evidence that vision enhances masked prediction performance during language processing, providing support that cross-modal representations in models can benefit individual modalities. However, we don't find evidence of brain-relevant information captured by the joint multi-modal transformer representations beyond that captured by all of the individual modalities. We finally show that the brain alignment of the pre-trained joint representation can be improved by fine-tuning using a task that requires vision-language inferences. Overall, our results paint an optimistic picture of the ability of multi-modal transformers to integrate vision and language in partially brain-relevant ways but also show that improving the brain alignment of these models may require new approaches

Large language models can segment narrative events similarly to humans

Humans perceive discrete events such as "restaurant visits" and "train rides" in their continuous experience. One important prerequisite for studying human event perception is the ability of researchers to quantify when one event ends and another begins. Typically, this information is derived by aggregating behavioral annotations from several observers. Here we present an alternative computational approach where event boundaries are derived using a large language model, GPT-3, instead of using human annotations. We demonstrate that GPT-3 can segment continuous narrative text into events. GPT-3-annotated events are significantly correlated with human event annotations. Furthermore, these GPT-derived annotations achieve a good approximation of the "consensus" solution (obtained by averaging across human annotations); the boundaries identified by GPT-3 are closer to the consensus, on average, than boundaries identified by individual human annotators. This finding suggests that GPT-3 provides a feasible solution for automated event annotations, and it demonstrates a further parallel between human cognition and prediction in large language models. In the future, GPT-3 may thereby help to elucidate the principles underlying human event perception

Getting aligned on representational alignment

Biological and artificial information processing systems form representations that they can use to categorize, reason, plan, navigate, and make decisions. How can we measure the extent to which the representations formed by these diverse systems agree? Do similarities in representations then translate into similar behavior? How can a system's representations be modified to better match those of another system? These questions pertaining to the study of representational alignment are at the heart of some of the most active research areas in cognitive science, neuroscience, and machine learning. For example, cognitive scientists measure the representational alignment of multiple individuals to identify shared cognitive priors, neuroscientists align fMRI responses from multiple individuals into a shared representational space for group-level analyses, and ML researchers distill knowledge from teacher models into student models by increasing their alignment. Unfortunately, there is limited knowledge transfer between research communities interested in representational alignment, so progress in one field often ends up being rediscovered independently in another. Thus, greater cross-field communication would be advantageous. To improve communication between these fields, we propose a unifying framework that can serve as a common language between researchers studying representational alignment. We survey the literature from all three fields and demonstrate how prior work fits into this framework. Finally, we lay out open problems in representational alignment where progress can benefit all three of these fields. We hope that our work can catalyze cross-disciplinary collaboration and accelerate progress for all communities studying and developing information processing systems. We note that this is a working paper and encourage readers to reach out with their suggestions for future revisions.Comment: Working paper, changes to be made in upcoming revision

Interpreting Deep Learning-Based Networking Systems

While many deep learning (DL)-based networking systems have demonstrated superior performance, the underlying Deep Neural Networks (DNNs) remain blackboxes and stay uninterpretable for network operators. The lack of interpretability makes DL-based networking systems prohibitive to deploy in practice. In this paper, we propose Metis, a framework that provides interpretability for two general categories of networking problems spanning local and global control. Accordingly, Metis introduces two different interpretation methods based on decision tree and hypergraph, where it converts DNN policies to interpretable rule-based controllers and highlight critical components based on analysis over hypergraph. We evaluate Metis over several state-of-the-art DL-based networking systems and show that Metis provides human-readable interpretations while preserving nearly no degradation in performance. We further present four concrete use cases of Metis, showcasing how Metis helps network operators to design, debug, deploy, and ad-hoc adjust DL-based networking systems.Comment: To appear at ACM SIGCOMM 202

Bridging Language in Machines with Language in the Brain

Several major innovations in artificial intelligence (AI) (e.g. convolutional neural networks, experience replay) are based on findings about the brain. However, the underlying brain findings took many years to first consolidate and many more to transfer to AI. Moreover, these findings were made using invasive methods in non human species. For brain functions that are uniquely human, such as understanding complex language, there is no suitable animal that can serve as a model organism and thus a mechanistic understanding is that much farther away. In this dissertation, we present a data-driven framework that circumvents these limitations by establishing a direct connection between brain recordings of people comprehending language and natural language processing (NLP) computer systems. We present evidence that this connection can be beneficial for both neurolinguistics and NLP. Specifically, we show that this framework can utilize recent successes in neural networks for NLP to enable scientific discovery about context- and task-dependent meaning composition in the brain, and we present the first evidence that brain activity measurements of people reading can be used to improve the generalization performance of a popular deep neural network language model. These investigations also contribute advances in cognitive modeling that may be useful beyond the study of language. In short, this dissertation involves multidisciplinary investigations and makes contributions to cognitive neuroscience, neurolinguistics, and natural language processing.</p

Plan estratégico y estudio de viabilidad del camping "El Maestrazgo"

Ivanova Toneva, M. (2014). Plan estratégico y estudio de viabilidad del camping "El Maestrazgo". http://hdl.handle.net/10251/44055.Archivo delegad

Local government debt in Spain: Spatial effects and budgetary stability

This doctoral thesis analyses a number of issues within the domain of local administration; issues such as local indebtedness, spatial effect on local debt and the law of financial sustainability and budgetary stability. The thesis is presented in three sections. After a brief introduction, the first section examines the indebtedness of Spanish town councils during the period 1997-2017, and the main legislative modifications introduced at both European and national level. A review of the main contributions made by the relevant literature on public debt, taken from different approaches and perspectives, is also included. The overall conclusion reached is that during the period prior to the onset of the economic crisis the level of indebtedness remained stable at different levels of Public Administration, with a clear downward trend. However, from 2008 onwards, the level of debt showed a sharp increase in growth until the Budget Stability Law of 2012 was imposed. This managed to slow down the growth of state and regional debt, thus achieving a downward trend in local debt. The second section of the thesis studies the local government debt, taking into account the presence of spatial interactions between neighbouring municipalities. For this, a sample of 527 municipalities of the Valencian Community is used and the spatial models S2SLS and Spatial lag are applied. The main results obtained show the existence of spatial autocorrelation in the outstanding debt among municipalities. The gender of the presiding mayor, subsidy index and transfers received and the average payment period are all variables which show a direct effect on outstanding debt. Furthermore, the variables index of net savings, the non-working population, local income and political strength have a direct effect on the municipality's debt and an indirect spatial effect on the debt of neighbouring municipalities. The third section aims to analyse whether the gender of the mayor and the rest of the members of the municipal council can affect compliance with the principles of budgetary stability and financial sustainability, as outlined in Organic Law 2/2012. The main results indicate that in local Spanish governments the greater presence of female mayors in local governance contributes to a better financial situation. In particular, it can be seen that a higher proportion of women working as councillors increases the probability of complying with budget stability, financial sustainability and the legal limit of indebtedness. Therefore, the Law on Effective Equality of Women and Men, which requires a minimum percentage of female participation in municipal electoral lists, favours compliance with the Budget Stability and Financial Sustainability Law.La presente tesis doctoral analiza diferentes temas de relevancia en el ámbito de las administraciones locales, tales como el endeudamiento local, los efectos espaciales en la deuda local y la ley de sostenibilidad financiera y la estabilidad presupuestaria. La tesis está compuesta por tres capítulos. Tras una breve introducción, en el primer capítulo se examina el endeudamiento en los ayuntamientos españoles durante el periodo de 1997 -2017, así como las principales modificaciones legislativas producidas a nivel europeo y nacional. También se estudian las principales aportaciones de la literatura que analizan la deuda pública, observadas desde diversos enfoques y perspectivas. Como conclusión principal se obtiene que, durante el periodo previo de la crisis económica, el nivel de endeudamiento se mantiene estable con una clara tendencia a la baja, en los diferentes niveles de las Administraciones Públicas. Es a partir del año 2008 cuando el nivel de deuda presenta un crecimiento elevado, hasta la entrada en vigor de la Ley de Estabilidad Presupuestaria del año 2012 que consigue suavizar el crecimiento de la deuda estatal y autonómica, logrando así una tendencia a la baja para la deuda local. El segundo capítulo de la tesis estudia la deuda de los gobiernos locales teniendo en cuenta la presencia de interacciones espaciales entre los municipios vecinos. Para ello se utiliza una muestra 527 municipios de la Comunidad Valenciana y se aplican los modelos espaciales S2SLS y Spatial lag. Los principales resultados obtenidos muestran la existencia de autocorrelacion espacial en la deuda viva entre los municipios. Las variables género del alcalde, índice de subvenciones y transferencias recibidas y el periodo medio de pago presentan efectos directos sobre la deuda viva. Mientras que, el índice de ahorro neto, la población no activa, la renta local y la fortaleza política tienen efectos directos en la deuda del municipio y efectos espaciales indirectos en la deuda de los municipios vecinos. El tercer capítulo tiene como objetivo analizar si el género del alcalde y el resto de miembros del consejo municipal pueden afectar el cumplimiento de los principios de estabilidad presupuestaria y sostenibilidad financiera, contenidos en la Ley Orgánica 2/2012. Los principales resultados apuntan que en los gobiernos locales españoles la mayor presencia de alcaldesas en la gobernabilidad local contribuye a mejorar la situación financiera de la entidad local. En particular, se observa que el trabajo de las mujeres con una alta proporción de concejalas mejora la probabilidad de cumplir con la estabilidad presupuestaria, la sostenibilidad financiera y el límite legal del endeudamiento. Por lo tanto, la Ley sobre la igualdad efectiva de mujeres y hombres, que exige un porcentaje mínimo de participación femenina en las listas electorales municipales, favorece el cumplimiento de la Ley de estabilidad presupuestaria y sostenibilidad financiera.Programa de Doctorat en Economia i Empres

Los límites legales al endeudamiento en los gobiernos locales

Treball final de Màster Universitari en Gestió Financera i Comptabilitat Avançada. Codi: SRL031. Curs acadèmic 2015-2016Este trabajo analiza el cumplimiento de los límites legales de deuda pública antes y después de la Ley de Estabilidad Presupuestaria del año 2012 en los municipios de la Comunidad Valenciana. El estudio se realiza durante el periodo de 2009 – 2013 mediante la utilización de la metodología tobit para variables acotadas. Los resultados obtenidos, indican en general, que la legislación ha incidido de manera positiva en el cumplimiento de las limitaciones. Asimismo, se aprecia que existen variables de diversa índole que afectan de forma positiva sobre el grado de cumplimiento de endeudamiento, como son la ideología, el indicador de estabilidad presupuestaria, gastos de capital per cápita, ahorro neto, capacidad fiscal y compromiso de gasto. Por el contrario, el resultado presupuestario no financiero presenta un impacto negativo sobre el grado de cumplimiento de limitaciones de endeudamiento

Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)

Neural network models for NLP are typically implemented without the explicit encoding of language rules and yet they are able to break one performance record after another. Despite much work, it is still unclear what the representations learned by these networks correspond to. We propose here a novel approach for interpreting neural networks that relies on the only processing system we have that does understand language: the human brain. We use brain imaging recordings of subjects reading complex natural text to interpret word and sequence embeddings from 4 recent NLP models - ELMo, USE, BERT and Transformer-XL. We study how their representations differ across layer depth, context length, and attention type. Our results reveal differences in the context-related representations across these models. Further, in the transformer models, we find an interaction between layer depth and context length, and between layer depth and attention type. We finally use the insights from the attention experiments to alter BERT: we remove the learned attention at shallow layers, and show that this manipulation improves performance on a wide range of syntactic tasks. Cognitive neuroscientists have already begun using NLP networks to study the brain, and this work closes the loop to allow the interaction between NLP and cognitive neuroscience to be a true cross-pollination